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Overview 



• Experimental Objectives: 

- To examine the effects of different command and control interfaces on UAS 
pilots' ability to respond to ATC commands and traffic advisories/warnings 

• What happens when a pilot who is operating "on-the-loop" (i.e., waypoint to 
waypoint/flight plan mode) needs to quickly get "in-the-loop" to respond to ATC 
clearance or traffic advisories? 

• Experimental Design: 

- 3 (Control Mode) X 3 (Event Type) Within-Subjects Factorial 

- Control Mode: 

1) Waypoint only 

2) Autopilot 

3) Manual 

- Event Type: 

1) ATC Clearance 

2) Well Clear Violation 

3) Resolution Advisory 


Overview 



Data Collection: 

- Dates: 8 JUL- 2 AUG 2013 

- Location: Flight Deck Display 
Research Lab (FDDRL) at NASA ARC 

Simulation Environment 

- Vigilant Spirit Control Station (VSCS; 
AFRL/RH) 

- Cockpit Situation Display (CSD) 

- SAA Processor 

- Multi Aircraft Control Station 
(MACS) 

• Airspace and air traffic 
environment 

• Pseudo pilot stations 

• Air Traffic Control (ATC) Stations 



Air Traffic Control Station 





GCS Configuration 
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UAS Ground Control Station Configuration 



Vigilant Spirit Control Station (AFRL/RH). Distribution A: Approved for public 
release; distribution unlimited. 88ABW Cleared 3/18/2013; 88ABW-2013-1303 





Scenarios 




• Scenarios: 

• Derived from FAA CONOPS 
scenarios (combination of "Loiter 
for Surveillance" and "Grid 
Pattern") 

• Class A & E Oakland Center 
Airspace (ZOA 40/41) with IFR and 
VFR traffic 

• UAS started at FL190, descended 
to 6000 ft to conduct a stepped 
grid pattern search, climbed back 
to FL190 

• Events were generated to force 
pilots to make quick control 
inputs: 

- ATC Clearances 


- Self-Separation Violations 

- Resolution (Collision) Advisories 


Participants 

• Participants: 

— 15 RQ-4 pilots Average age = 34 years old 

• 6/15 qualified through RQ-4 Basic Training (AF Specialty Code 18X) 

- Not required to have been previously qualified in a manned AC 

• 9/15 qualified through Undergraduate Pilot Training 

— Previously qualified in a manned AC 

• 9/15 had previous experience flying UAS in civil airspace 

- Average = 98 hours 

• All had Military Combat and/or Non-Combat experience 

- Average = 323 combined hours 

— 1 retired Air Traffic Controller with experience in Oakland 
Center airspace (confederate) 





Pilot Tasks 



• Pilot Task: 

• Operate a simulated MQ-1 
(HAWK21) along a pre-filed 
flight path within Oakland 
Center airspace under 
Instrument Flight Rules 

• Responsible only for air vehicle 
navigation (no sensor 
operation) 

• Comply with ATC clearances for 
traffic and/or weather as 
necessary 

• Respond to collision avoidance 
Resolution Advisories 





VSCS Control Mode 


• Primary Independent Variable: VSCS Control Mode 

1) Waypoint-to-Waypoint Mode (Waypoint; WP) (Baseline) 

• Functionality: can only change heading by modifying existing waypoints, can use override 
to change altitude 

2) Autopilot Mode (Autopilot; AP) 

• Retains WP functionality 

• Additional functionality: can change heading and altitude using new graphical interface 

3) Manual Mode (Manual; M) 

• Retains WP functionality 

• Additional functionality: can change heading and altitude using stick and throttle inputs 

> Pilots were able to use any method available to them to 
implement an edit 

— E.g., in Autopilot mode, the pilot could perform a vertical maneuver 
via waypoint edits or edits to the auto-pilot interface 


Waypoint Mode 
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Edit Plan Commands 


Apply+upload... Cancel 


Click and drag interface on TSD to 
move or add waypoints 


Right click or double 
click waypoint to 
open editing window 


Override capability for 
altitude and/or airspeed 


Altitude and airspeed inputs 
in editing window 


> Edits made via Waypoint Window 






Autopilot Mode 



Right click or double 
click ownship to open 
steering window and 
change mode to AP 



Steer HAWK21 


Override 


ft Override 


Override 


lets Override 


Altimeter 


Cancel 


Compass rose has drag-able 
heading bug and heading 
and altitude spinners 


Heading, altitude and speed 
holds can be input to the 
steering window interface 
via keyboard or spinners 


> Edits made via Steering Window or Compass Rose 








Manual Mode 





Switch to manual mode 
via steering window or 
on HOTAS button 


> Edits made via Stick and Throttle 




Event Type 





Somnrlnrv Inrirnrnrlrnt Ynrinhlr- Event Type 

1) ATC Clearances (ATC only): 

Traffic Alert during level flight 
Traffic Alert during climb/descent 
ATC vector for severe weather 

2) Self-Separation Viola tion (ATC & Display): 

Traffic Advisory (ATC) 




• Alert to future Well Clear Violation (Display) 
3) Resolution Advisory (Display only) 


Type and size of events were not experimentally controlled or 
counterbalanced across participants or scenarios 

• Clearances were up to discretion of controller, as permitted by the scenario 



Initial Research Question 



• What was the effect of the three different VSCS control modes 
on pilots' ability to comply with ATC clearances? 

— Pilot performance can best be understood by assessing their 
'Measured Response' (MR) 

• MR has been analyzed before by breaking down ATC-Pilot interactions into 
discrete stages (Shively, Vu & Baker, 2013) 

— Measured response data were analyzed utilizing a 2-Way Analysis 
of Variance (ANOVA) 

• Also measured (but not reported here): 

— Number of Uploads 
— Correctness 
— Size of Maneuver 
— Post Trial & Post Sim Questionnaires 



Stages of ATC-Pilot Interaction 




Stage 

Description 

Example (Left Turn in WP Mode) 

Source of Time Stamp 

ATC 


To 

Initial ATC 
Transmission 

"HAWK21, turn left heading 1-2-0, vectors 
for your descent." 

Voice Log - End of Relevant 
Controller Transmission 

Initiates 


Ti 

Pilot Reply 

"Turn left heading 1-2-0, HAWK21." 

Voice Log - Start of Relevant Pilot 
Transmission 

TIME 


t 2 

Pilot Initiates Edit 

Pilot opens Waypoint Window 

VSCS Camtasia - Moment 
Relevant Waypoint or Steering 
Window appears on display 



T 3a 

Pilot Uploads 1 st 
Edit 

Pilot incorrectly uploads 110° Hdg to the 
aircraft 

VSCS Camtasia & VSCS Output - 
Upload of First Relevant Edit 

Maneuver 


T 3b 

Pilot Uploads Final 
Edit 

Pilot correctly uploads 120° Hdg to the 
aircraft 

VSCS Camtasia & VSCS Output - 
Upload of Final and Correct Edit 

Completed 

> 

f 

t 4 

UAS Completes 
Maneuver 

HAWK21 reaches an acceptable range for 
the given clearance (120° Hdg, +/- 5°) 

VSCS Camtasia - UA Reaches 
Acceptable Range 




Stages of ATC-Pilot Interaction 
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T 0 

Ti 

t 2 

T 3 a 

T 3b 

t 4 

ATC: 

Pilot: 

Pilot: 

Pilot: 

Pilot: 

UAS: 

"Turn Left 
1-2-0" 

"Roger. Left 1-2-0" 

Opens Edit Window 

Uploads 1 st Edit 

Uploads Final Edit 

Completes 

Maneuver 


General Stats 



• Pilots were issued a total of: 

- 273 Traffic Advisories 

• Average of 6 advisories per trial 

• No action required; verbal response only 

- 767 Traffic Clearances 

• Average of 17 clearances per trial 

• By type: 

- Altitude Clearances: 229 

- Lateral Clearances: 300 

- 'Direct To' & 'Resume Own Nav' Clearances: 463 

• Immediate compliance expected 


> **Clearonce Type was not experimentally controlled** 


General Stats 



• Mode x Method Breakdown 

1) Waypoint = 270 total edits 

• All edits made via waypoint or steering window 

2) Autopilot = 253 total edits 

• 109 edits (43%) made via waypoint or steering window 

• 144 edits (57%) made via autopilot interface (Compass Rose) 

3) Manual = 244 total edits 

• 98 edits (40%) made via waypoint or steering window 

• 146 edits (60%) made via stick and throttle 

• Preliminary results include all edits made within a 
control mode, regardless of input method 


Preliminary Metrics 




Metric 

Calculation 

Description 

Verbal Response 
Time 

Ti - T 0 

Time it took for pilots to respond verbally to ATC advisories and clearances 

Initial Response 
Time 

T 2 - T 0 

Time it took for pilots to initiate edits in response to ATC clearances 

Initial Edit Time 
(1 st Upload) 

T 3a ' T 2 

Time it took pilots to upload their first edit from the moment they began editing 

Total Edit Time 
(Final Upload) 

T 3b " T 2 

Time it took pilots to upload their final edit from the moment they began editing 

Aircraft 
Response Time 

T 3a - T 0 

Time it took for the aircraft to begin maneuvering from ATC clearance 

Aircraft 

Maneuver Time 

^4 " T 3a 

Time it took the UAS to complete its maneuver once the maneuver began 

Compliance 

Time 

T 4 - To 

Time it took the UAS operator to complete all stages of ATC-Pilot interaction 


I 








ATC Clearance 
Ends 


Pilot 

Responds 


Pilot Initiates Pilot Makes Pilot Makes UAS Completes 

Edit 1 st Upload Final Upload Maneuver 



Preliminary Metrics 



Metric 

Calculation 

Description 

Verbal Response 
Time 

Ti - T 0 

Time it took for pilots to respond verbally to ATC advisories and clearances 

Initial Response 
Time 

T 2 - T 0 

Time it took for pilots to initiate edits in response to ATC clearances 

Initial Edit Time 
(1 st Upload) 

T 3a ' T 2 

Time it took pilots to upload their first edit from the moment they began editing 

Total Edit Time 
(Final Upload) 

^3b " ^2 

Time it took pilots to upload their final edit from the moment they began editing 

Aircraft 
Response Time 

T 3a - T 0 

Time it took for the aircraft to begin maneuvering from ATC clearance 

Aircraft 

Maneuver Time 

^4 " T 3a 

Time it took the UAS to complete its maneuver once the maneuver began 

Compliance 

T 4 - T 0 

Time it took the UAS operator to complete all stages of ATC-Pilot interaction 
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ATC Clearance 

Pilot 

Pilot Initiates 

Pilot Makes 

Pilot Makes 

UAS Completes 

Ends 

Responds 

Edit 

1 st Upload 

Final Upload 

Maneuver 



w Verbal Response Time (T ± - T 0 ) 


Time from ATC 
clearance to 
Pilot verbal 
response 


• Autopilot resulted in significantly shorter response times than Waypoint (p<.05) 

- No other differences were significant 

• Pilots replied to a total of 1,009 advisories & clearances 

— Waypoint = 1.47 sec 

- Autopilot = 1.18 sec 

- Manual = 1.40 sec 

> Grand Mean = 1.35 sec 



WP WP+AP WP+M 

Control Mode 



Verbal Response Time (T 1 - T 0 ) 



• Distribution: 

- 50% of participants replied 2 seconds or sooner following the 
controller's clearance 

- 90% of participants replied 3 seconds or sooner following the 
controller's clearance 


Preliminary Metrics 




Metric 

Calculation 

Description 

Verbal Response 
Time 

Ti - T 0 

Time it took for pilots to respond verbally to ATC advisories and clearances 

Initial Response 
Time 

T 2 ' T 0 

Time it took for pilots to initiate edits in response to ATC clearances 

Initial Edit Time 
(1 st Upload) 

T 3 a ' T 2 

Time it took pilots to upload their first edit from the moment they began editing 

Total Edit Time 
(Final Upload) 

^ 3 b " ^2 

Time it took pilots to upload their final edit from the moment they began editing 

Aircraft 
Response Time 

T 3a - T 0 

Time it took for the aircraft to begin maneuvering from ATC clearance 

Aircraft 

Maneuver Time 

^4 " T 3a 

Time it took the UAS to complete its maneuver once the maneuver began 

Compliance 

Time 

T 4 - T 0 

Time it took the UAS operator to complete all stages of ATC-Pilot interaction 
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Pilot 
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Edit 

Pilot Makes 
1 st Upload 

Pilot Makes 
Final Upload 

UAS Completes 
Maneuver 




Initial Response Time (T 2 - T 0 ) 



Time from ATC 
clearance to 
Pilot initial 
control input 



WP WP+AP WP+M 

Control Mode 


• Autopilot resulted in significantly shorter Initial Response Times than Waypoint 
and Manual (p<.05) 

• Pilots initiated a total of 549 edits in response to ATC clearances 

- Waypoint = 5.82 sec 

- Autopilot = 3.00 sec 

- Manual = 5.66 sec 

> Grand Mean = 4.83 sec 
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Initial Response Times (seconds) 


• Distribution: 

- 50% of participants started their edit 5 seconds or sooner following 
the controller's clearance 

- 90% of participants started their edit at 11 seconds or sooner 
following the controller's clearance 



Preliminary Metrics 




Metric 

Calculation 

Description 

Verbal Response 
Time 

T, - T 0 

Time it took for pilots to respond verbally to ATC advisories and clearances 

Initial Response 
Time 

T 2 - T 0 

Time it took for pilots to initiate edits in response to ATC clearances 

Initial Edit Time 
(1 st Upload) 

T 3a ■ T 2 

Time it took pilots to upload their first edit from the moment they began 
editing 

Total Edit Time 
(Final Upload) 

^3b " ^2 

Time it took pilots to upload their final edit from the moment they began editing 

Aircraft 
Response Time 

T 3a - T 0 

Time it took for the aircraft to begin maneuvering from ATC clearance 

Aircraft 

Maneuver Time 

^4 " T 3a 

Time it took the UAS to complete its maneuver once the maneuver began 

Compliance 

Time 

T 4 - T 0 

Time it took the UAS operator to complete all stages of ATC-Pilot interaction 



Pilot Initiates Pilot Makes 

Edit 1 st Upload 



Initial Edit Time (T 3a -T 2 ) 



Time from Pilot 
initial control 
input to first 
upload 



• Manual and autopilot resulted in significantly shorter times than Waypoint 

(P<-01) 

• Pilots successfully uploaded a total of 545 edits 

- Waypoint = 11.81 sec 

- Autopilot = 8.77 sec 

- Manual = 6.90 sec 

> Grand Mean = 9.16 sec 



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 More 

Initial Edit Times (seconds) 

• Distribution: 

- 50% of participants uploaded their initial edit 8 seconds or sooner 
following the start of their edit 

- 90% of participants uploaded their initial edit 18 seconds or sooner 
following the start of their edit 


Preliminary Metrics 




Metric 

Calculation 

Description 

Verbal Response 
Time 

T, - T 0 

Time it took for pilots to respond verbally to ATC advisories and clearances 

Initial Response 
Time 

T 2 - T 0 

Time it took for pilots to initiate edits in response to ATC clearances 

Initial Edit Time 
(1 st Upload) 

T 3a ' T 2 

Time it took pilots to upload their first edit from the moment they began editing 

Total Edit Time 
(Final Upload) 

T3b " ^2 

Time it took pilots to upload their final edit from the moment they began 
editing 

Aircraft 
Response Time 

T 3a - T 0 

Time it took for the aircraft to begin maneuvering from ATC clearance 

Aircraft 

Maneuver Time 

T 4 " T 3a 

Time it took the UAS to complete its maneuver once the maneuver began 

Compliance 

Time 

T 4 - T 0 

Time it took the UAS operator to complete all stages of ATC-Pilot interaction 


I 

I 

| « 

1 

— *| 

I 

1 

To 

1 

Ti 

t 2 

T 3 a 

T 3b 

1 

t 4 

ATC Clearance 
Ends 

Pilot 

Responds 

Pilot Initiates 
Edit 

Pilot Makes 
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Total Edit Time (T 3b - T 2 ) 
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Control Mode 
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• Manual and Autopilot resulted in significantly short times than Waypoint 
(p<.05) 

• Pilots successfully completed a total of 545 edits 

- Waypoint = 19.27 sec 

- Autopilot = 9.15 sec 

- Manual = 7.28 sec 

> Grand Mean = 11.90 sec 


Total Edit Time (T 3b - T 2 ) 



• Distribution: 

- 50% of participants uploaded their//na/ edit 8 seconds or sooner 
following the start of their edit 

- 90% of participants uploaded their//na/ edit 23 seconds or sooner 
following the start of their edit 


Preliminary Metrics 




Metric 

Calculation 

Description 

Verbal Response 
Time 

T, - T 0 

Time it took for pilots to respond verbally to ATC advisories and clearances 

Initial Response 
Time 

T 2 - T 0 

Time it took for pilots to initiate edits in response to ATC clearances 

Initial Edit Time 
(1 st Upload) 

T 3a ' T 2 

Time it took pilots to upload their first edit from the moment they began editing 

Total Edit Time 
(Final Upload) 

^3b " ^2 

Time it took pilots to upload their final edit from the moment they began editing 

Aircraft 
Response Time 

T 3a - T 0 

Time it took for the aircraft to begin maneuvering from ATC clearance 

Aircraft 

Maneuver Time 

^4 " T 3a 

Time it took the UAS to complete its maneuver once the maneuver began 

Compliance 

Time 

T 4 - T 0 

Time it took the UAS operator to complete all stages of ATC-Pilot interaction 
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Ti 

T 2 

T 3a 

T 3b 

t 4 

ATC Clearance 
Ends 

Pilot 

Responds 

Pilot Initiates 
Edit 

Pilot Makes 
1 st Upload 

Pilot Makes 
Final Upload 

UAS Completes 
Maneuver 



w Aircraft Response Time (T 3a 



Aircraft 

Response Time 



WP WP+AP WP+M 

Control Mode 


• Manual and Autopilot resulted in significantly shorter times than 
Waypoint (p<.01) 

• Pilots successfully completed a total of 602 clearances 

- Waypoint = 17.82 sec 

- Autopilot = 11.77 sec 

- Manual = 10. 05 sec 

> Grand Mean = 13.17 sec 



Aircraft Response Time (T 3a -T 0 ) 




Distribution: 


- 50% of participants started the AC maneuver within 12 seconds of the 
controller's clearance 

- 90% of participants started the AC maneuver within 25 seconds of the 
controller's clearance upload 


Preliminary Metrics 




Metric 

Calculation 

Description 




Initial Response 
Time 

O 

E- 

CN 

Time it took for pilots to initiate edits in response to ATC clearances 




Total Edit Time 
(Final Upload) 

T 3b " ^2 

Time it took pilots to upload their final edit from the moment they began editing 




Aircraft 

Maneuver Time 

^4 ■ T 3a 

Time it took the UAS to complete its maneuver once the maneuver began 






Pilot Makes Pilot Makes UAS Completes 

1 st Upload Final Upload Maneuver 


w Aircraft Maneuver Time (T 4 - T 3a ) 


Time for 
aircraft to 
complete 
maneuver from 
initial upload 



• No differences were found to be significant 

• Pilots successfully completed a total of 583 clearances 

- Waypoint = 56.14 sec 

- Autopilot = 59.48 sec 

- Manual = 49.94 sec 

> Grand Mean = 55.18 sec 


Aircraft Maneuver Time (T 4 -T 3a ) 



Distribution: 

- 50% of participants completed their maneuver 26 seconds or sooner 
following their initial upload 

- 90% of participants completed their maneuver 98 seconds or sooner 
following their initial upload 


Preliminary Metrics 



Metric 

Calculation 

Description 




Initial Response 
Time 

^2 " T 0 

Time it took for pilots to initiate edits in response to ATC clearances 




Total Edit Time 
(Final Upload) 

T 3b " ^2 

Time it took pilots to upload their final edit from the moment they began editing 




Aircraft 

Maneuver Time 

^4 " T 3a 

Time it took the UAS to complete its maneuver once the maneuver began 

Compliance 

Time 

T 4 - T 0 

Time it took the UAS operator to complete all stages of ATC-Pilot interaction 



ATC Clearance Pilot Pilot Initiates Pilot Makes Pilot Makes UAS Completes 

Ends Responds Edit 1 st Upload Final Upload Maneuver 


Compliance Time (T 4 - T 0 ) 


Time from ATC 
clearance to 
complete 
maneuver 


100 
90 
80 
70 

w 60 

§ 50 

a* * 

« 40 

30 
20 
10 
0 

WP WP+AP WP+M 

Control Mode 



• Manual resulted in significantly shorter compliance times than Waypoint (p<.05) 

— No other differences were significant 

❖ Expect significant results between WP + AP and WP when magnitude, method and dimension of maneuver are 
accounted for, based on time to initiate and final upload results 

• Pilots successfully completed a total of 571 clearances 

- Waypoint = 73.99 sec 

- Auto Pilot = 71.58 sec 

- Manual = 56.95 sec 

> Grand Mean = 67.51 sec 



Compliance Time (T 4 - T 0 ) 
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• Distribution: 

- 50% of participants completed their maneuver 40 seconds or sooner 
following the controller's clearance 

- 90% of participants completed their maneuver 120 seconds or sooner 
following the controller's clearance 


Results Summary 


• First phase of analysis examined the effect of three control 
modes on pilots' ability to comply with ATC clearances 

• The baseline condition (waypoint) showed significantly poorer 
performance in all but one of the metrics analyzed 

• Autopilot had significantly shorter Verbal Response and Initial 
Response times 

— Initial response times were almost twice as long for Waypoint and 
Manual 

• Both Manual and Autopilot had significantly shorter Edit and 
Aircraft Response Times 

— Total edit times were up to 12 seconds shorter then Waypoint 

— Aircraft response times were up to 8 seconds shorter than Waypoint 

• Manual had significantly shorter Compliance Times 


Results Summary 


• Takeaway: 

- The earliest stages of interaction, i.e. getting "in-the-loop," saw an 
advantage for the Autopilot mode 

- Both Autopilot and Manual saw substantial advantages in editing 
times 

• Manual mode was on average a few seconds faster the Autopilot because no 
edits were required (nav mode change only) 

- The limitations of the Waypoint mode is most apparent in its edit time 
(up to 12 sec slower); could have significant operational impact 

- Need to support pilots' ability to easily get in the loop to respond to 
ATC Clearances and SAA System alerts 

- Provide easy method for inputting holds - either through a manual or 
electronic interface - that are consistent with ATC and SAA system 
expectations/requirements 

- Waypoint to waypoint only interface may not be sufficient 


Results Summary 




• Caveats: 

— Preliminary data only 

• Lateral vs horizontal vs "direct to waypoint" inputs should be analyzed by control mode (some 
modes support different dimensions better) 

• Magnitude of maneuver needs to be accounted for 

- Tradeoff between experimental control and realistic, dynamic environment: 

1. Pilots had the freedom to use whichever method available (within a control mode) for a given 
clearance 

2. Type and number of clearances were not controlled or counterbalanced across participants or 
scenarios 

- Not every stage of interaction (TO - T4) was completed for each event 

• Ex: if a pilot was already in Manual mode and given a heading change, the only stages captured 
were T3b and T4 (start and end of maneuver) 

- Availability of override functionality in waypoint only mode closely resembles an 
AP or "quick input" functionality 

❖ Data are a result of one instantiation of a single prototype GCS 
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